Skip to content

Pass -gpu= at link time so the Thrust fatbin keeps every architecture - #17

Merged
Sophia Wen (hfwen0502) merged 1 commit into
mainfrom
fix-thrust-fatbin-arch-at-link
Sep 3, 2026
Merged

Pass -gpu= at link time so the Thrust fatbin keeps every architecture#17
Sophia Wen (hfwen0502) merged 1 commit into
mainfrom
fix-thrust-fatbin-arch-at-link

Conversation

@hfwen0502

@hfwen0502 Sophia Wen (hfwen0502) commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Problem

The Thrust extension passes -gpu=<arch> when compiling but not when linking.

That is not a cosmetic omission. The device-link step is where the final SASS is generated — nvc++ keeps device IR in the object file and codegens at link. So without -gpu= there, the architecture requested at compile time is discarded entirely and the artifact gets whatever the toolchain's own default target is. No warning, exit 0.

Measured on NVHPC 26.1 (aarch64, a cc100 host):

compile flag link flag resulting .so
(none) (none) sm_100
-gpu=cc90 (none)today's behaviour sm_100 ← asked cc90, got sm_100
-gpu=cc90 -gpu=cc90 — this PR sm_90

Note the middle row: it is not that a comma-separated list gets truncated, it is that any explicit request is overridden. A single-arch cc90 build is equally affected whenever the build host's default differs — which is why this surfaced as a Thrust-only failure on H100 from a build that had asked for the right architecture. Nothing masks it: these builds embed no PTX, so there is no JIT fallback.

_core_gpu_omp_offload was never affected — its extension already passes -gpu= in extra_link_args. That asymmetry is exactly why the symptom looked backend-specific rather than like a build-flag bug.

Fix

extra_link_args=extra_link_args + ['-mp', '-cuda', f'-gpu={gpu_arch}', '-lcudart'],

One line, matching what the OMP-offload extension already does.

Effect on main

main resolves the arch as _resolve_gpu_arch(default='cc90'), and the README documents that default. With this PR that default becomes effective rather than nominal: an unset SBD_GPU_ARCH now genuinely produces an sm_90 binary, as documented. Today it produces whatever the build host defaults to, which may not be sm_90 at all.

So this PR makes main's behaviour match main's README. It deliberately changes nothing else — no README edits, no change to the default.

Verification on a real build of this package

Multi-arch, on GB200:

build _core_gpu_thrust SASS size
before sm_100 10 M
after sm_80 sm_90 sm_100 16 M

Still numerically correct — run_sbd_diag.py, h2o-1em3, 4 ranks (2×2):

--device gpu      Energy = -76.23586672306634
--device gpu-omp  Energy = -76.23586672306634

matching the published reference for that determinant set.

Notes

  • The kept architecture comes from the toolchain, not from probing hardware: building on a GPU-less host (a container build stage) does not change it.
  • -gpu=ccall-major also works on the Thrust path, yielding sm_80 sm_90 sm_100 sm_110 sm_120 — a reasonable choice for distributed images instead of enumerating.

Follow-ups (deliberately not in this PR)

  1. Assert after build_ext that the architectures in the linked extension match SBD_GPU_ARCH (cuobjdump --list-elf), so a silent arch drop fails the build instead of surfacing on someone else's GPU months later.
  2. Making SBD_GPU_ARCH optional — unset meaning "target the build host", set meaning "honor exactly, including several generations" — is implemented on fix-conda-build-and-mpi-detection and will arrive on main with that branch at the next PyPI release.

The device-link step is where the final SASS is generated, so an arch passed
only at compile time is discarded and the extension silently gets nvc++'s own
default target instead of what was requested.

Measured on NVHPC 26.1 (aarch64), compile with -gpu=cc90:
  .so linked without -gpu=  ->  sm_100   (the host default, not cc90)
  .so linked with    -gpu=  ->  sm_90

_core_gpu_omp_offload already passes -gpu= at link, which is why only the Thrust
backend was affected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@hfwen0502
Sophia Wen (hfwen0502) force-pushed the fix-thrust-fatbin-arch-at-link branch from 1b85c87 to ae463c8 Compare September 3, 2026 20:30
@hfwen0502
Sophia Wen (hfwen0502) merged commit 6c436c9 into main Sep 3, 2026
6 checks passed
@hfwen0502
Sophia Wen (hfwen0502) deleted the fix-thrust-fatbin-arch-at-link branch September 3, 2026 20:41
Sophia Wen (hfwen0502) added a commit that referenced this pull request Sep 11, 2026
Squashes the 22 commits of fix-conda-build-and-mpi-detection, rebased onto
main so that #15, #16, #17 and #21 are preserved rather than reverted.

setup.py
- Derive MPI_HOME from the MPI mpi4py is actually linked against, instead of
  guessing, so the extension and mpi4py cannot disagree at runtime.
- Make conda-based builds work: strip the gcc-only tokens RHEL 9 CPython bakes
  into sysconfig that nvc++ rejects, and drop conda's -B compiler_compat while
  keeping its -L/-rpath entries.
- Make SBD_GPU_ARCH optional and honor it at link as well as compile, so a
  multi-arch Thrust fatbin keeps every architecture it was asked for. This
  supersedes the narrower fix in #17.

python/__init__.py, device_config.py
- Load the backends lazily, one per process, and build all three by default.
  Removes the silent GPU demotion and the need for a second install.
- Report why a backend is missing, and fail instead of silently using the CPU
  when gpu-omp has no device.

README: rewritten around the conda recipe and split GPU prerequisites by build
vs run. Retains #15's point that the sdist bundles the headers.
Sophia Wen (hfwen0502) added a commit that referenced this pull request Sep 11, 2026
* setup: derive MPI from mpi4py, fix conda builds, load backends lazily

Squashes the 22 commits of fix-conda-build-and-mpi-detection, rebased onto
main so that #15, #16, #17 and #21 are preserved rather than reverted.

setup.py
- Derive MPI_HOME from the MPI mpi4py is actually linked against, instead of
  guessing, so the extension and mpi4py cannot disagree at runtime.
- Make conda-based builds work: strip the gcc-only tokens RHEL 9 CPython bakes
  into sysconfig that nvc++ rejects, and drop conda's -B compiler_compat while
  keeping its -L/-rpath entries.
- Make SBD_GPU_ARCH optional and honor it at link as well as compile, so a
  multi-arch Thrust fatbin keeps every architecture it was asked for. This
  supersedes the narrower fix in #17.

python/__init__.py, device_config.py
- Load the backends lazily, one per process, and build all three by default.
  Removes the silent GPU demotion and the need for a second install.
- Report why a backend is missing, and fail instead of silently using the CPU
  when gpu-omp has no device.

README: rewritten around the conda recipe and split GPU prerequisites by build
vs run. Retains #15's point that the sdist bundles the headers.

* Add AMD GPU support to the OpenMP target-offload backend

Squashes the four commits of amd-rocm-omp-offload.

The same _core_gpu_omp_offload source now builds under either vendor: nvc++
-mp=gpu for NVIDIA, amdclang++ --offload-arch=gfx* for AMD. Thrust stays
NVIDIA-only, since upstream wires it to nvc++ -cuda and there is no rocThrust
configuration to build; SBD_BUILD_BACKEND=gpu therefore fails on AMD rather
than silently falling back to the CPU.

setup.py
- detect_gpu_toolchain() picks nvidia or amd, with SBD_GPU_VENDOR to force the
  choice on a host carrying both; find_rocm_toolchain() honors ROCM_HOME then
  PATH, mirroring find_nvidia_hpc_sdk().
- SBD_GPU_ARCH is spelled per vendor and applied at compile and link: nvc++
  takes -gpu=cc80,cc90 as one flag, clang needs a repeated --offload-arch=.
  Unset on AMD it is detected with the amdgpu-arch that ships beside the chosen
  compiler (not one found on PATH, which can come from a different ROCm), and a
  GPU-less build host is a hard error rather than an unrunnable module.
- amdclang++ needs far less sysconfig scrubbing than nvc++: only
  -fcf-protection, and -march=x86-64-v2 is deliberately left alone.

python/
- 'gpu-omp' is vendor-neutral; get_backend('gpu-omp').__sbd_offload_target__
  reports what an install was actually built for, and the Thrust backend is
  stamped the same way. rocm/gpu-amd-omp/gpu-rocm-omp are aliases.
- device_config.auto() resolves against the backends actually COMPILED instead
  of returning 'gpu' whenever any GPU was detected, which selected the CUDA
  Thrust backend on AMD hosts. GPU presence is probed with rocm-smi as well as
  nvidia-smi.
- bindings.cpp pins each rank to one offload device, reading the vendor's
  device-visibility variable when omp_get_num_devices() reports 0 under dlopen.

README: one rule for MPI -- every GPU backend hands MPI device pointers, so use
a GPU-aware MPI (CUDA-aware on NVIDIA, ROCm-aware on AMD).

Verified on MI250X (gfx90a, ROCm): 8 ranks, 4x2 grid, --device gpu-omp.

* Pin the offload device in gdb_diag too

sbd_pin_offload_device() was wired into tpb_diag and tpb_diag_from_files but
not gdb_diag, so with --device gpu-omp every rank ran gdb_diag on the default
device -- the "all ranks land on GPU 0" symptom, for that entry point only.

Not caught earlier because the three call sites are copies rather than one
helper; folding them together is a separate cleanup.

* Tighten GPU detection and honor ROCM_PATH

- Per-probe timeouts in _gpu_available(): rocm-smi keeps the 30 s it needs to
  enumerate a multi-GCD node, but nvidia-smi no longer inherits it. A wedged
  NVIDIA driver hangs rather than failing, and it was stalling backend
  selection for 30 s instead of 5.

- Require a GPU[<n>] line from `rocm-smi --showid` rather than trusting its
  exit status, in both _gpu_available() and DeviceConfig._check_hip(). A ROCm
  install on a GPU-less BUILD host is exactly where a bare `rocm-smi` exiting 0
  claims hardware that is not there, after which auto() picks a backend that
  cannot run. This is the same signal get_device_info() already counts devices
  with, so there is now one convention instead of two.

- find_rocm_toolchain() honors ROCM_PATH as well as ROCM_HOME, checked in that
  order so an explicit ROCM_HOME still wins. ROCM_PATH is the variable ROCm
  itself sets, and tox.ini already passed it through to a build that ignored it.

* macOS: take libomp and OpenBLAS from the conda env

The Darwin block hardcoded Homebrew paths, so a conda env carrying llvm-openmp
still compiled against /opt/homebrew -- while at import time the process loads
the conda copies anyway, resolved through the interpreter's rpath. Prefer
$CONDA_PREFIX when it has include/omp.h; Homebrew stays as the fallback.

Two diagnostics, since this was hard to see:
- fail with the fix when neither source has libomp, rather than "'omp.h' file
  not found" from the middle of a compile
- print the resolved compiler and its version. sysconfig records a bare
  clang++, which PATH resolves, so a Homebrew LLVM silently wins over both
  Apple clang and a conda toolchain.

README: llvm-openmp in both conda recipes, a note to pin CC/CXX, and a
troubleshooting entry for a conda compiler package shadowing nvc++/amdclang++.

Verified on macOS arm64 under Apple clang 21.0.0 and conda clang 23.1.0: no
Homebrew references in the linked extension, pytest and the 2-rank MPI test
green in both environments.

* Restore the mpicc fallback for MPI detection

Deriving the MPI prefix from mpi4py dropped main's mpicc probe, which broke CI:
Debian splits Open MPI's headers to /usr/lib/<triple>/openmpi/include, outside
the prefix, so mpi4py names the right MPI but $prefix/include/mpi.h does not
exist and the build gave up.

A prefix without mpi.h is an unusual layout, not the wrong MPI, so ask mpicc
before failing -- both when the prefix lacks the header and when no prefix is
found at all. mpi4py stays the preferred source, so the extension and mpi4py
still cannot diverge.

* Make the mpicc probe work for MPICH as well as Open MPI

--showme:compile is an Open MPI-ism. MPICH's wrapper treats it as a source file
and tries to compile it (exit 127 here), so probing that first silently ruled out
MPICH -- which this package claims to support.

Probe `-show` first, which both wrappers understand and which prints the whole
command line, and keep --showme as the second attempt. Duplicate -I/-L entries
are collapsed (MPICH repeats them), and the libraries the wrapper names are used
as-is: -lmpi for Open MPI, -lmpi -lpmpi for MPICH.

Verified against conda-forge openmpi 5.0.10 and mpich 5.0.1.

* Probe mpicc with -show only

-show is understood by every wrapper checked -- homebrew open-mpi, conda-forge
openmpi 5.0.10, conda-forge mpich 5.0.1 -- while --showme:compile works on the
Open MPI ones and exits 127 on MPICH. So -show strictly dominates, and the second
attempt was dead weight: one subprocess call instead of up to three.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant